Evil Dataset Proof of Concept (updated proposal) #31
Conversation
There was a problem hiding this comment.
Hey there! For context, when I leave comments in a PR I typically tag them as [blocking] or [non-blocking].
[blocking] comments are things I personally believe are important enough that I want to see them discussed/resolved before approving the change.
[non-blocking] comments are things I think would be nice to discuss/resolve, but not the end of the world if they are left as-is.
Obviously, feel free to disregard if you aren't seeking my specific signoff! :-)
|
|
||
| ``` | ||
| datapoint-library/ | ||
| ├─ example-datapoint/ |
There was a problem hiding this comment.
[blocker] Happy to iterate on the design as appropriate, but I'm curious to hear your thoughts on the number of "datapoints" you think we'll have in the medium and long term and whether we'll want to stick with this approach of separate directory structures for each of them? Or are we thinking of making a custom file format that encompasses all the data for a "datapoint"? Or maybe multiple "datapoints"?
| > curl -s -XGET 'https://localhost:9200/_search?pretty' -ku "admin:admin" -H "Content-Type: application/x-ndjson" --data-binary @query.json | jq -f filter.jq | diff - expected.json | ||
|
|
||
| # An unsuccesful comparison might look like the following: | ||
| > curl -s -XGET 'https://localhost:9200/_search?pretty' -ku "admin:admin" -H "Content-Type: application/x-ndjson" --data-binary @query.json | jq -f filter.jq | diff - expected.json |
There was a problem hiding this comment.
[Blocker] Trying to understand your longer-term intent here and what the scripting implications will be. Do you see us using curl directly for the foreseeable future? Or are we going to use the client SDKs for whatever language the test script is written in?
| @@ -0,0 +1,78 @@ | |||
| # Evil Dataset | |||
|
|
|||
| This directory is intended to house the "evil dataset"--a collection of data & associated queries that test edge cases and behavior changes between versions. | |||
There was a problem hiding this comment.
[Blocker] Just trying to understand your intent. Are you envisioning this housing all of our "expectations"?
Description
This PR is a proof of concept for the updated proposal for the evil dataset (#9). It adds a single, fairly trivial example, along with the documentation/code samples to run it.
Testing
Testing has been manual so far, and this is just POC for the time being, so I think that's sufficient until we've settled on a path forward.
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.